Parallel processing using multi-core processor

ABSTRACT

Disclosed are methods, systems, paradigms and structures for processing data packets in a communication network by a multi-core network processor. The network processor includes a plurality of multi-threaded core processors and special purpose processors for processing the data packets atomically, and in parallel. An ingress module of the network processor stores the incoming data packets in the memory and adds them to an input queue. The network processor processes a data packet by performing a set of network operations on the data packet in a single thread of a core processor. The special purpose processors perform a subset of the set of network operations on the data packet atomically. An egress module retrieves the processed data packets from a plurality of output queues based on a quality of service (QoS) associated with the output queues, and forwards the data packets towards their destination addresses.

CLAIM OF PRIORITY

This is a continuation of U.S. patent application Ser. No. 13/786,306,filed on Mar. 5, 2013, which claims the benefit of U.S. ProvisionalPatent Application No. 61/740,374, filed on Dec. 20, 2012, both of whichare incorporated herein by reference in its entirety.

FIELD OF INVENTION

This invention generally relates to parallel processing. Morespecifically, the invention relates to parallel processing usingmulti-core processors.

BACKGROUND

A multi-core processor is a single computing component with two or moreindependent actual central processing units (called “cores”), which arethe units that read and execute program instructions. Multi-coreprocessors are used across many application domains includinggeneral-purpose, embedded, network, digital signal processing (DSP), andgraphics. The improvement in performance gained by the use of amulti-core processor depends very much on the software algorithms usedand their implementation. In particular, possible gains are limited bythe fraction of the software that can be run in parallel simultaneouslyon multiple cores.

In multi-core network processing, a key challenge is how to exploit allthe cores in these devices to achieve maximum networking performance,despite the performance limitations inherent in a symmetricmultiprocessing operating system. In the current multi-core networkprocessor architecture, the processing of a data packet is done by oneor more core processors. The synchronization between various coreprocessors and various threads of core processors, memory management,etc. are achieved through software programming. Further, when theprocessing of the data packet passes from one core to another core, thestate or synchronization information is passed between the cores orbetween the threads of the core processing the data packet.

Significant resources are consumed in passing the state/synchronizationinformation between the threads or the processors which reduces theavailability of the resources for performing actual network applicationrelated tasks. Therefore, the overhead of passing thestate/synchronization information decreases the efficiency of theprocessor.

Further, in current architecture, the core processor does not performany useful network related operation while waiting for the arrival ofthe next data packet. The core processor may be idle until the next datapacket arrives at the core processor. In current systems that havemulti-core processors with an array of pipelined special purpose cores,the multi-core processor is limited by the capabilities of the specialcores and pipeline depth. The multi-core is also limited by the slowestcore.

SUMMARY

Introduced here are methods, systems, paradigms and structures forparallel processing using multi-core processors. The disclosed solutioncan be used in various applications such as general purpose processing,network processing, storage systems, etc. In one embodiment, the abovemulti-core processor may be used as a (network) processor in acommunication network for processing data packets. The network processorincludes a plurality of multi-threaded core processors and a pluralityof special purpose processors for processing the data packets atomicallyand in parallel. The network processor processes a data packet byperforming a set of network operations on the data packet. The set ofnetwork operations can be based on applications/network protocols suchas Resource Reservation Protocol (RSVP).

A particular data packet is processed by a single core processor and bya single thread of the core processor. The core processor may requestone or more special purpose processors to perform a subset of the set ofnetwork operations on the data packet. A special purpose processor isany processing element that exclusively handles all accesses and updatesto a specific set of shared resources such that those shared resourcesare always accessed and updated atomically. The special purposeprocessor may be hardware specifically designed to handle certain typesof state data (for example, meters, queues, statistics, Volume Table ofContents (VTOCs), etc.) or general purpose cores which have beendedicated to processing the specified set of shared resources. The oneor more special-purpose processors processes the data packet byperforming the set of operations atomically. That is, the networkprocessor ensures that the shared resources of the network processorused for processing the particular data packet is not used by otherresources of the network processor while the particular data packet isbeing processed. This locking out of the shared resources involved inthe process ensures the atomicity of the process.

An ingress module in the network processor allocates memory for theincoming data packets and stores the data packets in a memory of thenetwork processor. The ingress module generates a packet buffer chainlinking various buffers containing portions of the data packet, for adata packet whose size exceeds the buffer size. The ingress module addsthe incoming data packets to an input queue from which one of the coreprocessors retrieves a data packet for further processing. An outputpacket processor in the network processor adds the processed data packetto one of a plurality of output queues based on a quality of service(QoS) associated with the output queue and forwards the data packetscontained in the output queues towards their destination addresses basedon the QoS.

Since a single thread of the core processor performs all the set ofnetwork operations on the data packet, no state information orsynchronization information is passed between processors, which resultsin faster and more efficient processing of data packets. Also, thenumber of memory reads and writes are minimized reducing the delaycaused due to memory reads and writes. The discussed method, systems,structures and paradigms increase a number of effective operations perpacket (EOPP), that is, the number of instructions that can be performedon the data packet to perform application related tasks.

Some embodiments of the invention have other aspects, elements,features, and steps in addition to or in place of what is describedabove. These potential additions and replacements are describedthroughout the rest of the specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example network processor in which an embodiment of theinvention may operate.

FIG. 2 is a flow diagram illustrating a process of processing datapackets by a network processor.

FIG. 3 is a flow diagram illustrating a process for performing a set ofnetwork operations on a data packet in a single thread of a coreprocessor.

FIG. 4 is a flow diagram illustrating a process of processing datapackets per RSVP (RFC 2475) application/networking protocol in thenetwork processor of FIG. 1.

FIG. 5 is a flow diagram illustrating a process of processing input datapackets.

FIG. 6 is a block diagram illustrating an example of a structure forrepresenting data packets stored in a memory of a network processor ofFIG. 1.

FIG. 7 is a flow diagram illustrating a process of storing the datapacket in a memory of the network processor of FIG. 1.

FIG. 8 is a flow diagram of process for adding a data packet to an inputqueue of a network processor of FIG. 1.

FIG. 9 is a block diagram of an output queue structure of a networkprocessor of FIG. 1.

FIG. 10 is a flow diagram illustrating a process of processing outputdata packets.

FIG. 11 is a block diagram illustrating an egress module of a networkprocessor of FIG. 1.

DETAILED DESCRIPTION

References in this description to “an embodiment”, “one embodiment”, orthe like, mean that the particular feature, function, or characteristicbeing described is included in at least one embodiment of the presentinvention. Occurrences of such phrases in this specification do notnecessarily all refer to the same embodiment, nor are they necessarilymutually exclusive.

Introduced here are methods, systems, paradigms and structures forparallel processing using multi-core processors. The disclosed solutioncan be used in various applications such as general purpose processing,network processing, storage systems, etc. In one embodiment, the abovemulti-core processor may be used as a network processor in acommunication network for processing data packets. The network processorprocesses a data packet by performing a set of network operations on thedata packet. The set of network operations can be based onapplications/network protocols such as Resource Reservation Protocol(RSVP).

A particular data packet is processed by a single core processor and bya single thread of the core processor. The core processor may requestone or more special purpose processors to perform a subset of the set ofnetwork operations on the data packet. The one or more special purposeprocessors processes the data packet by performing the set of operationsatomically. That is, the network processor ensures that the sharedresources of the network processor used for processing the particulardata packet is not used by other resources of the network processorwhile the particular data packet is being processed.

An ingress module in the network processor allocates memory for theincoming data packets and stores the data packets in a memory of thenetwork processor. The ingress module generates a packet buffer chainlinking various buffers containing portions of the data packet, for adata packet whose size exceeds the buffer size. The ingress module addsthe incoming data packets to an input queue from which one of the coreprocessors retrieves a data packet for further processing. An outputpacket processor in the network processor adds the processed data packetto one of a plurality of output queues based on a quality of service(QoS) associated with the output queue and forwards the data packetscontained in the output queues towards their destination addresses basedon the QoS.

Since a single thread of the core processor performs all the set ofnetwork operations on the data packet, no state information orsynchronization information is passed between processors, which resultsin faster and more efficient processing of data packets. Also, thenumber of memory reads and writes are minimized reducing the delaycaused due to memory reads and writes. The discussed method, systems,structures and paradigms increase a number of effective operations perpacket (EOPP), that is, the number of instructions that can be performedon the data packet to perform application related tasks.

FIG. 1 is an example network processor 100 in which an embodiment of theinvention may operate. Data packets from various source computer systems(not shown) are transmitted to destination computer systems (not shown)via a communication network (not shown). The communication networkincludes various systems such as routers (not shown) that facilitateforwarding the data packets towards their corresponding destinationaddresses. The routers perform operations including receiving the datapackets, performing error checking on the received data packets, findingnext hop addresses for the data packets, fragmenting the data packetsetc. before the packets are forwarded to their corresponding next hopaddresses. In an embodiment, the network processor 100 may be used insuch routers to perform the above discussed operations.

The network processor 100 includes a plurality of multi-threaded coreprocessors such as core processor 105, a plurality of special purposeprocessors such as special purpose processor 115, a memory such ascontrol memory 120 and data memory 125, an ingress module 130 to receiveand store incoming data packets in the memory, and an egress module 135to forward the processed data packets towards their destinationaddresses. An internal command network 110 such as a crossbar switchenables communication between various components in the networkprocessor 100.

Each of the core processors processes data packets. However, a datapacket is processed by a single core processor, such as core processor105, and by a single thread of the core processor 105. The coreprocessor 105 processes the data packet by performing a set of networkoperations on the data packet. The core processor 105 may request one ormore special purpose processors to perform a subset of the set ofnetwork operations on the data packet. The one or more special purposeprocessors processes the data packet by performing the subset of the setof network operations atomically. That is, the network processor 100ensures that the shared resources of the network processor 100 used forprocessing the particular data packet is not used by other resources ofthe network processor 100 while the particular data packet is beingprocessed by the core processor 105 or the special purpose processor115.

The core processors and special purpose processors perform the set ofnetwork operations in parallel. The set of network operations performedon the data packet can be based on applications/network protocols suchas Resource Reservation Protocol (RSVP). In an embodiment, the RSVP canbe a protocol defined by Request For Comment (RFC) 2475. The set ofnetwork operations in RSVP (RFC 2475) can include receiving datapackets, performing error handling, performing table look up operations,metering, generating statics, policing or congestion control, forwardingdata packets towards their destination addresses etc.

In an embodiment, the special purpose processors may perform frequentlyexecuted tasks such as table lookup for finding next hop address for thedata packet, metering, policing, congestion control etc. The coreprocessors may perform tasks such as error checking, overseeing theoperations performed by the special purpose processors etc. In anembodiment, the network processor 100 may have one special purposeprocessor for each of the identified network operations. However, inanother embodiment, the network processor 100 may have a special purposeprocessor execute more than one network operation. The network processor100 ensures that certain operations on the data packets are performedatomically by using a semaphore associated with a thread requesting anoperation on the data packet.

A semaphore is used for signaling a target processor such as a coreprocessor or a special purpose processor when a request has beenaccomplished and any data is safe to use. For example, a semaphoreattached with a table update operation may be tested by the coreprocessor 105 to determine if the results of the table lookup have beenreturned to the core processor 105. The thread on the core processor 105may continue doing other tasks after sending any request(s) to one ormore special purpose processors, until the thread needs to use theresults of the request(s). Before using the results of any request, thecore processor 105 will test the semaphore that was attached to thespecific request. If the request has been completed and the results havebeen returned to the core processor 105, then the thread will continueprocessing. If the request has not been completed then the coreprocessor 105 will put the thread in sleep mode when the semaphore istested, and will switch to processing other threads. When the semaphoreis returned, the core processor 105 will wake the thread and the threadwill continue processing as if the semaphore had been available at thetime of the test.

The core processors obtain the data packets from an input queue of thespecial purpose processor such as input packet queue processor 115. Thecore processor 105 may request the input packet queue processor 115 toreturn a data packet. The requests from the core processors are added toa request queue, and are served on a First-In-First-Out (FIFO) basis.The request queue may have requests from various threads of each of thecore processors. For example, if there are four core processors in thenetwork processor 100 and each of the four core processors can executethree threads, the request queue can have up to twelve (4×3) requests inthe request queue.

The ingress module 130 performs memory management operations including(i) allocating memory to incoming data packets, and (ii) storing theincoming data packets in the memory. The ingress module 130 stores thedata packets in a memory such as data memory 125. Further, the ingressmodule 130 may store pointers to the data packets in control memory 120.In an embodiment, access to the data packets is provided by controlmemory 120. The data packets are stored in the buffers of data memory125. Each of the buffers is of a configurable size. If the size of thedata packet exceeds the size of the buffer, the ingress module 130splits the data packet and stores portions of the data packet in aplurality of buffers. The ingress module 130 generates a data packetbuffer chain for the data packet. The data packet buffer chain is apointer structure that links buffers containing various portions of thedata packet. After the data packet buffer chain is created, the datapacket is added to the input queue by adding a pointer to the datapacket buffer chain. If the input queue already contains data packets,the data packet buffer chain of the last data packet in the input queueis linked to the data packet buffer chain of the newly added datapacket.

The input packet queue processor 115 returns a data packet from the headof the input queue to a thread that is first in the request queue. Ifthere are no data packets in the input queue, the requesting thread isswitched to sleep mode until a data packet is available to be returnedto the thread. When a data packet is returned to a thread of the coreprocessor 105, the core processor 105 performs the set of networkoperations on the data packet in the single thread. The core processor105 may request one or more special purpose processors to perform asubset of the network operations atomically. While the special purposeprocessors are performing the operations atomically, the thread of thecore processor 105 which requested the special purpose processor may beswitched to sleep mode until a result of execution by the specialpurpose processor is posted back to the thread. Putting this coreprocessor into sleep mode at this point avoids semaphore wait cycles andenables the other processors to more efficiently concentrate onprocessing background tasks.

After the data packet is processed, the core processor 105 requests aspecial purpose processor such as an output packet queue processor 140to add the processed data packet to one of a plurality of output queues.The output packet queue processor 140 adds the data packet to aparticular output queue based on the QoS contracted to the data packet.An egress module 135 retrieves the data packets from the output queuesbased on the QoS of the output queues and forwards the data packetstowards their corresponding destination addresses. After the data packetis forwarded to the next hop address, the memory allocated to the datapacket is released or freed by the output packet queue processor 140.

In an embodiment, the control memory 120 and data memory 125 may be onthe same or different physical memory devices in the network processor100. The control memory 120 has a list of all the buffers in the datamemory 125. The list in the control memory 120 can be a one-to-onemapping to the buffers in the data memory 125. In an embodiment, theone-to-one mapping can include pointers to buffers of the data memory125. The one-to-one mapping in control memory 120 has a pointer to anygiven address of a data buffer.

The control memory 120 or data memory 125 represents any form of randomaccess memory (RAM), flash memory, content addressable memory (CAM) suchas ternary CAM, or the like, or a combination of such devices. In use,the control memory 120 may contain data structures such as packet bufferchain, output packet queue header structures and data structures havingcommands such as memory read or memory write that are executed often.

The internal command network 110 can be a distributed crossbar switch ordata network or semaphore network that facilitates communication betweenvarious components such as core processors, special purpose processors,control memory, data memory, etc. in the network processor 100. Theinternal command network 110 takes input from the core processors andoutputs it to a particular destination. For example, if a core processorwants to write data to the control memory 120, the input can be the dataand the destination can be control memory 120. The internal commandnetwork 110 contains a plurality of switches, wherein each of the coreprocessors is connected to a series of switches that would connect aparticular core processor to a particular destination.

The network processor 100 executes various operations such as networkoperations, memory management, etc. using special purpose hardwareprocessors. Further, since all of the network operations on the datapacket are performed in a single thread of a core processor, there is nooverhead of passing state information or synchronization informationbetween various stages of processing. Also, the memory management isperformed by hardware modules such as ingress module 130, input packetqueue processor 115, output packet queue processor 140, etc.Accordingly, the performance offered by the network processor 100 issignificantly increased. Further, the programming model to execute theset of network operations can be a single threaded programming modelwhich is less complex than multi-threaded programming.

FIG. 2 is a flow diagram illustrating a process 200 of processing datapackets by a network processor, according to an embodiment of thedisclosed technique. The process 200 may be executed in a networkprocessor such as network processor 100 of FIG. 1. At step 205, thenetwork processor 100 receives a plurality of data packets from one ormore source computer systems in a computer network. At step 210, thenetwork processor 100 performs a set of network operations on each ofthe data packets. The set of network operations performed can be basedon a particular application/networking protocol. Each of the datapackets are processed in a single thread of a specific core processor ofthe network processor 100. At step 215, the network processor 100forwards the processed data packets towards their correspondingdestination addresses based on a QoS contracted to the data packets.

FIG. 3 is a flow diagram illustrating a process 300 for performing a setof network operations on a data packet in a single thread of a coreprocessor, according to an embodiment of the disclosed technique. Theprocess 300 may be executed in a network processor such as networkprocessor 100 of FIG. 1. Certain network operations are performed by acore processor and certain by special purpose processors of the networkprocessor 100. In an embodiment, frequently executed operations may beperformed by special purpose processors. For example, in a networkprocessor 100 executing networking protocol such as RSVP (RFC 2475),network operations such as metering, policing, generating statisticsetc. can be executed by special purpose processors.

At step 305, a core processor such as the core processor 105 requeststhe special purpose processors to execute a subset of the set of networkoperations atomically. The core processor 105 attaches a semaphore to athread of the core processor 105 requesting the operation. At step 310,the special purpose processor performs the subset of the set of networkoperations on the data packet atomically. At step 315, the specialpurpose processor posts the results of the execution of the operationsto a specified memory address.

Referring back to step 310, performing the set of operations atomicallyincludes ensuring that while a shared resource of the network processor100 is being updated by a particular resource, other resources are notallowed to access the shared resource. The semaphore attached to thethread of the core processor 105 switches the thread to a sleep modeuntil the special purpose processor has completed executing theoperation on the data packet and posted the results of the execution toa specified memory address. After the special purpose processor hascompleted executing the operation, the semaphore wakes up the thread ofthe core processor and the rest of the network operations are continued.In an embodiment, if the thread of the core processor is not dependenton the results of execution of the operations by the special purposeprocessor, the thread may not be switched to sleep mode. The thread maycontinue processing the next set of operations on the data packet.

FIG. 4 is a flow diagram illustrating a process 400 of processing datapackets using RSVP (RFC 2475) application/networking protocol, accordingto an embodiment of the disclosed technique. The process 400 may beexecuted in a network processor such as network processor 100 of FIG. 1.At step 402, a thread of a core processor such as core processor 105sends a request to an input packet queue processor 115 to obtain a datapacket. The request also includes a semaphore associated with thethread. If the request queue of the input packet queue processor 115already has requests from other threads of the core processor 105 orother core processors, the request from the specific thread of the coreprocessor 105 is added to the request queue on a FIFO basis.

At determination step 404, the core processor 105 determines whether thesemaphore has indicated that the input packet queue processor 115returned the data packet to the thread. Responsive to a determinationthat the input packet queue processor 115 has not returned the datapacket to the thread, at step 406, the thread is switched to sleep modeuntil a data packet is returned to the thread. On the other hand,responsive to a determination that the input packet queue processor 115has returned the data packet to the thread, the core processor 105continues processing the data packet at step 408. In an embodiment, whenthe data packet is returned to the core processor 105, the input queuepacket processor 115 returns a pointer to the data packet stored in thedata memory 125 of the network processor 100. Further, a portion of thedata packet, such as a header of the data packet, is read from the datamemory 125 and written into a local memory of the core processor 105.The core processor 105 may refer to the portion stored in the localmemory while processing the data packet.

At determination step 408, the core processor 105 determines whetherthere are any errors in the data packet. The errors detected include,for example, size of the data packet. If the size of the data packet islesser than a predefined minimum data packet size, the core processor105 determines that the data packet is an erroneous data packet. Thecore processor 105 may also check for other errors by computing achecksum for the data packet. Initially, when the packet arrives, theingress module 130 computes the checksum to validate whether the datapacket was received correctly. The checksum may be modifiedcorresponding to any modifications to the data packet by the coreprocessor 105. Responsive to a determination that the data packet is anerroneous data packet, at step 410, the core processor 105 sends arequest to the output packet queue processor 140 to drop the datapacket. On the other hand, responsive to the determination that the datapacket is not an erroneous data packet, the core processor 105 continuesprocessing the data packet at step 412.

At step 412, the core processor 105 sends a request to a special purposeprocessor to perform a table lookup operation to determine the next hopaddress for the data packet. The request also includes a semaphoreassociated with the thread. At step 414, the special purpose processorperforms the table lookup operations to determine the next hop addressfor the data packet and posts the results to a specified memory address.The network processor 100 supports both IPv4 and IPv6, that is, IPversions 4 and 6. Also, the special purpose processor may also determinea queue identification (ID) of an output queue, to which the data packetmay be added, based on a QoS contracted to the data packet. In anembodiment, the table lookup operations can also include reverse pathvalidation and microflow parameters. Microflow is a single instance ofan application-to-application flow of packets which is identified bysource address, source port, destination address, destination port andprotocol id. The next hop address can include an Internet Protocol (IP)address.

At determination step 416, the core processor 105 determines whether thesemaphore has indicated that table lookup operations have executed andresults have been posted to a specific location in the memory.Responsive to a determination that the results have not been posted yet,at step 418, the thread of the core processor 105 is switched to a sleepmode until the results are posted. On the other hand, responsive to adetermination that the table lookup results have been posted, the coreprocessor 105 continues processing the data packet at step 420.

At step 420, the core processor 105 sends a request to a special purposeprocessor for performing metering operations. The request also includesa semaphore associated with the thread of the core processor 105. In anembodiment, metering operations, per RFC 2475, can include measuring thetemporal properties (e.g., rate) of a traffic stream selected by aclassifier (which identifies particular data packets). The instantaneousstate of this process may be to affect the operation of a marker,shaper, or dropper, and/or may be used for accounting, measurement orcongestion control purposes.

At step 422, the special purpose processor performs the meteringoperations for the data packet. At determination step 424, the coreprocessor 105 determines whether the semaphore indicates that themetering operations have executed and results have been posted.Responsive to a determination that the metering results have not beenposted, at step 426, the thread of the core processor 105 is switched tosleep mode until the results are posted. On the other hand, responsiveto a determination that the metering operations have completed executingand the results are posted, the core processor 105 continues processingthe data packet at step 428.

At step 428, the core processor 105 sends a request to a special purposeprocessor to perform congestion control or policing operations on thedata packet. In an embodiment, the congestion control, per RFC 2475, isa process of discarding packets within a traffic stream in accordancewith the state of a corresponding meter enforcing a traffic profile. Thedata packets may be dropped based on the data obtained from the meteringoperations. The request from the core processor 105 also includes asemaphore associated with the thread of the core processor 105requesting the special purpose processor.

At step 430, the congestion control operations are performed by thespecial purpose processor. At determination step 432, the core processor105 determines whether the semaphore has indicated that the congestioncontrol process has executed and posted the results. Responsive to adetermination that the results of the congestion control operations arenot yet available, at step 434, the thread of the core processor 105 isswitched to sleep mode until the results are posted. On the other handresponsive to a determination that the congestion control results havebeen posted, the core processor 105 continues processing the data packetat step 436.

At determination step 436, the core processor 105 determines if aportion of the data packet, such as the header of the data packet storedin the local memory of the core processor 105, has been modified by anyof the set of network operations. In an embodiment, certain operationssuch as fragmenting the data packet, updating the checksum, time to live(TTL) for the data packet etc. modify certain portions of the datapacket. Responsive to a determination that the data packet is notmodified, the control is transferred to step 446. On the other hand,responsive to a determination that the portion of the data packet ismodified, at step 438, the core processor 105 sends a request to aspecial purpose processor to write the modified portion of the datapacket from the local memory of the core processor 105 to the datamemory 125. The core processor 105 also includes a semaphore with therequest.

At step 440, the special purpose processor executes the memory writeoperation to write the modified portions of the data packet to the datamemory 125. At determination step 442, the core processor 105 determineswhether the semaphore has indicated that the memory write operation hasexecuted and the modified portion of the data packet has been written tothe data memory 125. Responsive to a determination that the memory writeoperation has not yet completed writing the modified portion of the datapacket, the thread of the core processor 105 is switched to sleep modeuntil the modified portion is written to the data memory 125. On theother hand, responsive to a determination that the modified portion hasbeen written to the data memory 125, the core processor 105 continuesprocessing the data packet at step 446.

At step 446, the core processor 105 sends a request to the output packetqueue processor 140 to add the data packet to an output queue. The coreprocessor 105 also includes a semaphore with the request. At step 448,the output packet queue processor 140 executes queuing operations to addthe data packet to an appropriate output queue. In an embodiment, theoutput packet queue processor 140 identifies the output queue based onthe queue ID of the data packet determined during the table lookupoperations.

At determination step 450, the core processor 105 determines whether thesemaphore indicates that the data packet has been added to the outputqueue. Responsive to a determination that the data packet is not yetadded to the output queue, at step 452, the thread of the core processor105 is switched to sleep mode until the data packet is added to theoutput queue. On the other hand, responsive to a determination that thedata packet is added to the output queue, the core processor 105determines that the thread has completed processing the data packet. Atstep 454, the thread of the core processor 105 can start processing anext data packet by sending a request to the input packet queueprocessor 115.

Further, the egress module 135 obtains data packets from the outputqueues based on the QoS associated with the output queues and forwardsthe data packets towards their corresponding destination addresses.

Each of the core processors of the network processor 100 and each of thethreads of a particular processor can perform the above discussedoperations on different data packets in parallel.

Referring back to the operations discussed in FIG. 4, the networkprocessor 100 executes certain commands for every data packet that isprocessed by the network processor 100. Further, these commands aresimilar for every thread of the core processor. The network processor100 creates data structures to store the commands, and initializes thedata structures even before a first thread of a first processor requestsa data packet from the input queue processor. In an embodiment, thecommands include (a) memory reads—reading data (data packet header anddata packet descriptor) from data memory 125 to a local memory of a coreprocessor, (b) memory writes—writing modified data (data packet headerand data packet descriptor) from local memory of core processor to thedata memory, (c) adding the data packet to an output queue (d) updatingerror counters for the data packets, (e) table look up commands, (d)metering commands, (f) congestion control commands etc. Initializing orpre-allocating memory to the data structures for the above commands evenbefore the network processor 100 starts processing the first data packetand using the data structures for every thread of every core processorreduces the time consumed for performing these common administrationoperations for every data packet. The network processor 100 can utilizeits processing power on performing the network protocol related tasksand thus, increasing the number of EOPP. All of the strands in thenetwork processor may not be running ‘similar’ code.

Although it is possible for all threads to be running similar code, forexample, code for processing incoming data packets, in an embodiment,not every thread of a core processor 105 may be processing the datapackets. It is possible to program such that only a subset of thethreads are processing incoming packets. Other threads could beprogrammed to execute tasks, for example, background tasks, that wouldonly run when the higher priority threads (processing incoming datapackets) are in sleep mode.

The architecture of the network processor 100 allows the processes suchas (a) input data packet operations—receiving the input data packets andadding them to the input queue, (b) processing the data packets by thecore processors, and (c) output data packet operations—forwarding thedata packets from the output queues to their corresponding destinationaddresses, to execute independently. That is, for example, thearchitecture separates a requirement for when a data packet is receivedby a network processor 100 and when the data packet is processed. Thedata packet is added to the input queue when it arrives at the networkprocessor 100, and the core processor may retrieve the data packet fromthe input queue at a later time, for example, when the core processor isready, to process the data packet. Such an architecture allows thenetwork processor 100 to spend as much time as it is necessary orrequired for processing a data packet, for example, for performing deeppacket inspection, before moving on to the next one.

FIG. 5 is a flow diagram illustrating a process 500 of processing inputdata packets, according to an embodiment of the disclosed technique. Theprocess 500 may be executed in a network processor such as networkprocessor 100 of FIG. 1. At step 505, the ingress module 130 receives adata packet. The data packet may be received from various computers in acomputer network the network processor 100 is part of. At step 510, theingress module 130 (or a memory allocation module working in cooperationwith the ingress module 130) allocates memory to the received datapacket from, for example, data memory 125 of the network processor 100.At step 515, the data packet is stored in the data memory. At step 520,the ingress module 130 instructs the input packet queue processor 115 toadd the data packet to the input queue.

Referring back to step 515, the data packet is stored one or more of thebuffers of the data memory 125. The data memory 125 contains a pluralityof configurable size buffers. If the size of a received data packet islarger than the buffer size, the data packet is split into a number ofportions and each of the portions is stored in a buffer. The ingressmodule 130 (or a packet buffer chain generation unit working incooperation with the ingress module 130) links each of the buffershaving portions of the data packet to form a packet buffer chain for thedata packet.

Referring back to step 520, the network processor 100 ensures that thedata packet is accessible by the core processor 105 only after allportions of the data packet are written into the data memory 125. Theingress module 130 attaches a semaphore to memory write operations thatwrite the data packet to the memory. The input packet queue processor115 is does not send a data packet to the requesting core processoruntil all portions of the data packet are written to the data memory.When all the portions of the data packet are written, the semaphorewakes up the input packet queue processor 115 indicating that completedata packet is written into the memory. The input packet queue processor115 may then send the data packet to requesting core processor. The coreprocessor then pauses the background tasks and resumes its high-priorityfunction.

FIG. 6 is a block diagram illustrating an example 600 of a structure forrepresenting data packets stored in a memory of a network processor,according to an embodiment of the disclosed technique. The example 600may be implemented in a network processor such as network processor 100of FIG. 1. The data packets received by the ingress module 130 arestored in the data memory 125. In an embodiment, the control memory 120may have packet buffer structures that contain pointers to the datapackets stored in the data memory 125. For example, for a data packet“Packet 0” stored in the data memory 125, the control memory 120 maystore a packet buffer chain 605 containing packet buffer structures 605,610, and 615 that have pointers to the buffers containing data packet“Packet 0.” The network processor 100 provides access to the datapackets stored in the data memory 125 through the control memory 120using the above packet buffer structures.

If a size of the data packet is lesser than the size of the buffer ofthe data memory, the data packet is stored in a single buffer. If thedata packet is stored in a single buffer, the packet buffer chain forthe data packet may have only packet buffer structure. For example, thepacket buffer chain 625 for data packet “Packet 1” has only one packetbuffer structure. On the other hand, when a size of the data packet islarger than the size of the buffer, the data packet is split into anumber of portions and the portions are stored in a number of buffers.Each of the buffers containing portions of the data packet is linked toform a packet buffer chain such as packet buffer chain 605. A packetbuffer structure can include two pointers, a next data packet pointerfor pointing to a next data packet in the input queue, and a next bufferpointer for pointing to the next buffer of the same data packet. In somepacket buffer structures one or none of the pointers may be used.

In the example 600, for data packet “Packet 0,” the first packet bufferstructure 610 (also referred to as header packet buffer structure 610)of the packet buffer chain 605 includes a pointer “Pkt 1,0” that pointsto the next data packet “Packet 1” in the input queue, and the nextbuffer pointer “Pkt 0, 1” that points to the next buffer containing thenext portion of the data packet “Packet 0.” In the packet bufferstructure 615, however, only next buffer pointer “Pkt 0,2” is used. Thenext packet pointer is not used. The next packet pointer can have a nullvalue. In the last packet buffer structure 620 (also referred to as tailpacket buffer structure 620) of the packet buffer chain 605, none of thepointers are used. The next packet pointer can have a null value. Thenext buffer pointer can have a value, for example, end of packet (EOP),that indicates that the current buffer is the last buffer in the packetbuffer chain 605 for the data packet “Packet 0.”

The packet buffer structures and packet buffer chains are created whenthe data packets are stored in the data memory 125. When the data packetis added to the input queue, a pointer to the header buffer structure ofthe packet buffer chain of the data packet is added to the input queue.If the input queue already contains data packets, the next packetpointer in the header buffer structure of the last data packet isupdated to point to the header buffer structure of the data packet addedto the input queue. For example, when the data packet “Packet 1” isadded to the input queue, the next data packet pointer of the headerpacket buffer structure 610 is updated to point to the header bufferstructure of the data packet “Packet 1.”

When the core processor requests a data packet from the input packetqueue processor 115, the input packet queue processor 115 obtains thedata packet (pointer to the packet buffer structure of the data packet)from the head of the input queue and returns it to the core processor.In an embodiment, similar packet buffer structures may be used forqueuing the data packets in the output queues of the network processor100.

FIG. 7 is a flow diagram illustrating a process 700 of storing the datapacket in a memory of the network processor, according to an embodimentof the disclosed technique. The process 700 may be executed by a networkprocessor such as network processor 100 of FIG. 1. At determination step705, the ingress module 130 determines whether a size of the data packetis larger than the buffer size of the data memory 125 of the networkprocessor 100. Responsive to a determination that the data packet sizeis not larger than the buffer size, at step 710, the ingress module 130stores the data packet in a single buffer and creates a packet bufferstructure with a value indicating that the packet buffer structure isthe last buffer structure in the packet buffer chain for the datapacket.

On the other hand, responsive to a determination that the data packetsize is larger than the buffer size, at step 715, the ingress module 130stores the data packet in a number of buffers, each of the buffershaving a distinct portion of the data packet. At step 720, the ingressmodule 130 (or a packet buffer generation unit that works in cooperationwith the ingress module 130) generates a packet buffer structure foreach of the buffers storing a portion of the data packet. At step 725,the ingress module 130 links each of the packet buffer structures,starting from a header packet buffer structure corresponding to a firstbuffer containing a first portion the data packet to the tail bufferstructure corresponding to a last buffer containing the last portion ofthe data packet, to create a packet buffer chain for the data packet.

FIG. 8 is a flow diagram of process 800 for adding a data packet to aninput queue of a network processor, according to an embodiment of thedisclosed technique. The process 800 may be executed by a networkprocessor such as a network processor 100 of FIG. 1. Adding the datapacket to the input queue includes adding a pointer that points to theheader buffer structure of a packet buffer chain of a data packet to theinput queue. After writing the data packet to the memory, the ingressmodule 130 sends an instruction to the input packet queue processor 115to add the data packet to the input queue. At determination step 805,the input packet queue processor 115 determines whether the input queueis empty. Responsive to a determination that the input queue is empty,at step 810, the input packet queue processor 115 adds a pointer to theheader packet buffer structure of the packet buffer chain of the datapacket to the input queue, and the process 800 returns.

Responsive to a determination that the input queue is not empty, at step815, the input packet queue processor 115 obtains a header bufferstructure of the packet buffer chain of the last data packet in theinput queue. At step 820, the input packet queue processor 115 updatesthe next packet pointer of the header packet buffer structure of thelast data packet in the input queue to point to the header packet bufferstructure of the packet buffer chain of the data packet added to theinput queue.

FIG. 9 is a block diagram of an output queue structure 900 of a networkprocessor, according to an embodiment of the disclosed technique. Theoutput queue structure 900 may be implemented in a network processorsuch as network processor 100 of FIG. 1. The network processor 100forwards the data packets to their corresponding next hop addressesbased on the QoS contracted to the data packets. For example, videostreaming data may be guaranteed a pre-defined bandwidth and/or speed, aclass of senders may be guaranteed a pre-defined bandwidth and/or speed,email data may have lower priority than video data, etc. Accordingly,the video data may be forwarded before email data. The QoS contracted tothe data may be based on a number of policies. The output queuestructure 900 provides a number of output queue classes and outputqueues for queuing output data packets based on the QoS contracted tothe data packets. The egress module 135 fetches the data packets fromthe appropriate output queue classes and/or output queues based on theQoS contracted to the data packet and forwards them to their next hopaddresses.

The output queue structure 900 is a hierarchical structure of ports ofan output link of the network processor 100, output queue classes andoutput queues. Every output queue class and an output queue can beassociated with a specific QoS. A port 905 has a set of output queueclasses such as output queue class 910 and the output queue class 910has a set of output queues such as output queue 915. For example, anoutput queue class can have an output queue for each sender that is partof that output queue class. In an embodiment, an output queue such asoutput queue 920 can be directly linked to the port 905 of the outputlink instead of an output queue class.

Each of the output queue classes and/or output queues is allocated acertain amount of resource. For example, if for a given time period anoutput queue class is allocated five percent (5%) of the bandwidth andthe output queue class has five output queues in it, each of the outputqueues can be allocated one percent (1%) of the bandwidth, or four ofthe queues can be allocated one half percent (0.5%) of the bandwidth andone of the output queues may be allocated three percent (3%) of thebandwidth. The output queues and the output queue classes can be set upin number of ways based on the QoS contracts agreed with the senders.When the data packets are processed by the network processor, the amountof bandwidth consumed by each of the data packets is tracked, forexample, by metering operations of FIG. 4. If an amount of the resourceconsumed by a particular output queue class or an output queue exceedsthe pre-defined limit, a number of pre-defined actions may be performedon the data packets belonging to those output queue classes or outputqueues including dropping the data packets.

The data packet 925 is added to the output queue 915 based on a queue IDof the data packet 925, which is obtained, for example, during the tablelook up operations of FIG. 4. The queue ID identifies the output queueclass or the port to which the output queue of the data packet 925belongs. Accordingly, the output packet queue processor 140 adds thedata packet 925 to the appropriate queue 915.

FIG. 10 is a flow diagram illustrating a process 1000 of processingoutput data packets, according to an embodiment of the disclosedtechnique. The process 1000 may be executed in a network processor suchas network processor 100 of FIG. 1. At step 1005, the output packetqueue processor 140 receives a request from the core processor 105 toadd the data packet to an output queue. At step 1010, the output packetqueue processor 140 obtains the queue ID of the data packet. The queueID identifies the output queue class or the port to which the outputqueue of the data packet belongs. At step 1015, the output packet queueprocessor 140 adds the data packet to the identified output queue of theoutput queue structure. At step 1020, the egress module 135 obtains thedata packets from the output queues based on the QoS associated with theoutput queues and/or output queue classes. At step 1025, the egressmodule 135 forwards the data packets to their corresponding next-hopaddresses.

Referring back to step 1005, the output queue requests from various coreprocessors or various threads of a specific core processor are stored ina request queue of the output packet queue processor 140 on a FIFObasis.

Referring back to step 1015, in an embodiment, the output queuestructure 900 includes only output queue classes and/or output queuesthat contain data packets. The output queue structure 900 is not acomplete list of all output queue classes or output queues that can bedefined for the network processor 100. When a data packet is added to anoutput queue, if the output queue is empty, the output queue is added toan output queue class to which it belongs. If the output queue class isitself empty, that is, does not already exist in the output queuestructure 900, the output queue class is created in the output queuestructure 900 and the output queue is added to the output queue class.The data packet is then added to the output queue.

Referring back to step 1020, the process (steps 1005-1015) of adding thedata packets to the output queue structure 900 is independent of theprocess (steps 1020-1025) of forwarding the data packets towards theirnext hop destination. The egress module 135 may obtain the data packetsfrom the output queue structure 900 independent of when the data packetsare added to the output queue structure 900. That is, the data packetsadded to the output queue structure 900 may not be immediately forwardedto their next-hop addresses. The egress module 135 fetches the datapackets from the output queues or output queue classes based on the QoSassociated with the output queues or output queue classes. For example,if video data is guaranteed higher bandwidth than email data, the outputqueues or output queue classes containing video data packets may beserved first and the output queues or output queue classes containingemail data packets may be served later.

FIG. 11 is a block diagram of a system 1100 illustrating an egressmodule 1135 of a network processor, according to an embodiment of thedisclosed technique. The egress module 1135 can be similar to the egressmodule 135 of network processor 100 of FIG. 1. The core processorsrequest the output packet queue processor 140 to add the processed datapackets to the output queue structure. The queue requests can includeinformation regarding an output queue class and/or output queue to whichthe data packet belongs, and a pointer to header packet buffer structureof the data packet. The output packet queue processor 140 receives thequeue requests via internal command network 1105, and stores them in anoutput packet request queue 1110 on a FIFO basis.

A packet queue finite state machine module 1115 retrieves the queuerequests from the output packet request queue 1110 on FIFO basis. Afterretrieving a queue request, the packet queue finite state machine module1115 requests the control memory 1125 to read the output queue headerstructure/output queue class header structure of the output queue/outputqueue class into a queue cache 1120 for a given queue ID associated withthe data packet. Once the header structure is loaded, the packet queuefinite state machine module 1115 identifies the corresponding outputqueue and/or output queue class of the data packet, and adds the datapacket to the output queue.

In an embodiment, the network processor 100 can support up to a millionoutput queues or output queue classes. An output queue class or outputqueue header structure identifies a particular output queue or queueclass to which a data packet belongs, a weight of the output queue orqueue class, address to the next output queue or output queue class, abit that indicates whether the output queue belongs to a output queueclass or a port, etc. The header structures can be stored in controlmemory 1125 (similar to control memory 120 of FIG. 1) or in other memorysuch as data memory 125 from which the control memory 120 can fetch.

An egress scheduler 1130 retrieves the data packets from the outputqueues or output queue classes based on the QoS associated with them,and the egress module 1135 forwards the data packets to theircorresponding next-hop addresses via the output link of the networkprocessor 100. The output link of the network processor can be based onnetwork processor streaming interface (NPSI). For example, the egressscheduler 1130 looks at the output queue classes and/or the outputqueues using information about how much bandwidth has been allocated toeach of the output queues or output queue classes to do a fair roundrobin, or easily weighted round robin, to give priority to the outputqueue classes or output queues that have more bandwidth. But over aperiod of time, every output queue or output queue class would getprocessed if they have a certain amount of bandwidth allocated.

In cases where other queues have higher priority (QoS) such as voice orvideo, queues which are on a best-effort basis, such as email, wouldonly be processed when the higher-priority queue quotas have beensatisfied.

The techniques introduced herein can be implemented by, for example,programmable circuitry (e.g., one or more microprocessors) programmedwith software and/or firmware, or entirely in special-purpose hardwired(non-programmable) circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more ASICs, PLDs, FPGAs, etc.

Software or firmware for use in implementing the techniques introducedhere may be stored on a machine-readable storage medium and may beexecuted by one or more general-purpose or special-purpose programmablemicroprocessors. A “machine-readable storage medium”, as the term isused herein, includes any mechanism that can store information in a formaccessible by a machine.

A machine can also be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a laptop computer, a set-top box (STB), apersonal digital assistant (PDA), a cellular telephone, an iPhone, aBlackberry, a processor, a telephone, a web appliance, a network router,switch or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

A machine-accessible storage medium or a storage device(s) or a memoryincludes, for example, recordable/non-recordable media (e.g., ROM; RAM;magnetic disk storage media; optical storage media; flash memorydevices; etc.), etc., or any combination thereof. The storage mediumtypically may be non-transitory or include a non-transitory device. Inthis context, a non-transitory storage medium may include a device thatis tangible, meaning that the device has a concrete physical form,although the device may change its physical state. Thus, for example,non-transitory refers to a device remaining tangible despite this changein state.

The term “logic”, as used herein, can include, for example, programmablecircuitry programmed with specific software and/or firmware,special-purpose hardwired circuitry, or a combination thereof.

What is claimed is:
 1. A method comprising: receiving, at an outputpacket processor from one of a plurality of core processors in a networkprocessor, a request to add a data packet to an output queue structure,the output queue structure including a plurality of output queues, aplurality of queue classes and a plurality of ports to an output link ofthe network processor, each of the output queues belonging to (i) one ofthe queue classes or (ii) one of the ports, and each of the queueclasses belonging to one of the ports, and wherein each of the queueclasses or output queues is associated with a pre-defined QoS;obtaining, from the data packet, a queue identification (ID) of the datapacket; identifying, based on the queue ID, an output queue to which theparticular data packet belongs, the output queue belonging to one of theclasses or one of the ports; and adding the data packet to the outputqueue.
 2. The method of claim 1 further comprising: forwarding, by anegress module of the network processor and based on the QoS of (i) theoutput queues or (ii) queue classes, data packets contained in theoutput queue structure towards corresponding destination addresses ofthe data packets.
 3. The method of claim 1, wherein receiving therequest to add the data packet to an output queue structure includesreceiving a plurality of requests from the core processors to add aplurality of data packets to the output queue structure, and adding therequests to a request queue.
 4. The method of claim 2, wherein obtainingthe queue ID of the data packet includes requesting, by a packet queuemodule, a memory of the network processor to store the queue ID of thedata packet in a queue cache, and storing, by the memory, the queue IDin the queue cache.
 5. The method claim 4, wherein adding the datapacket to the output queue structure includes reading, by the packetqueue module, the queue ID from the queue cache, and adding, by thepacket queue module and based on the queue ID, the data packet to theoutput queue structure.
 6. The method of claim 2, wherein forwarding thedata packets contained in the output queue structure towards theircorresponding destination addresses includes retrieving, by an egressscheduler of the egress module and based on the QoS associated with thequeue classes or the output queues, the data packets from the outputqueue structure.
 7. A network processor comprising: an output packetqueue processor to receive, from one of a plurality of core processorsin the network processor, a queue command to add a data packet processedby the network processor to an output queue structure, the output packetprocessor configured to store the queue command in a command queue; apacket queue finite state machine module to add the data packet to theoutput queue structure based on a queue identification (ID) of the datapacket, the output queue structure including a plurality of outputqueues, a plurality of queue classes and a plurality of ports to anoutput link of the network processor, each of the output queuesbelonging to (i) one of the queue classes or (ii) one of the ports, andeach of the queue classes belonging to one of the ports, and whereineach of the queue classes or output queues is associated with apre-defined QoS; and an egress scheduler to retrieve data packetscontained in the output queue structure based on the QoS of (i) theoutput queues or (ii) queue classes, the egress scheduler configured toforward the data packets towards corresponding destination addresses ofthe data packets.